ConTextual Masked Auto-Encoder for Dense Passage Retrieval

Dense passage retrieval aims to retrieve the relevant passages of a query from large corpus based on dense representations (i.e., vectors) and passages. Recent studies have explored improving pre-trained language models boost performance. This paper proposes CoT-MAE (ConTextual Masked Auto-Encoder), simple yet effective generative pre-training method for retrieval. employs an asymmetric encoder-decoder architecture that learns compress sentence semantics into vector through self-supervised context-supervised masked auto-encoding. Precisely, auto-encoding model tokens inside text span, semantical correlation between spans. We conduct experiments large-scale benchmarks show considerable improvements over strong baselines, demonstrating high efficiency CoT-MAE. Our code is available at https://github.com/caskcsg/ir/tree/main/cotmae.